{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "NeNNkb1wT87m"
   },
   "source": [
    "# Tutorial: DSPy and LanceDB Integration"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "TjcJnSLZT87p"
   },
   "source": [
    "This tutorial demonstrates the integration of DSPy with LanceDB to create a scalable and efficient data processing and querying system. Each section will guide you through the key steps involved, with explanations provided for the corresponding blocks of code.\n",
    "\n",
    "### Introduction\n",
    "In this notebook, we integrate DSPy, a powerful data science library, with LanceDB, a high-performance database designed for machine learning applications. This combination is particularly effective for managing, processing, and querying large datasets in machine learning workflows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "W6kN5neAduOZ",
    "outputId": "4aee8de8-f337-4ce8-a8ae-f71f7f123e69"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m4.5/4.5 MB\u001b[0m \u001b[31m34.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25h"
     ]
    }
   ],
   "source": [
    "!pip install dspy lancedb torch tantivy -q"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "id": "j73CK2KFT87s"
   },
   "outputs": [],
   "source": [
    "import re\n",
    "import dspy\n",
    "import torch\n",
    "import lancedb\n",
    "\n",
    "from lancedb.embeddings import get_registry\n",
    "from lancedb.pydantic import LanceModel, Vector\n",
    "from lancedb.rerankers import (\n",
    "    LinearCombinationReranker,\n",
    ")  # LanceDB hybrid search uses LinearCombinationReranker by default"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "wHnH1_HqT87u"
   },
   "source": [
    "Sets up the device (GPU if available, otherwise CPU) and initializes the \"BAAI/bge-small-en-v1.5\" embedding model using Hugging Face."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 318,
     "referenced_widgets": [
      "dcaa0ac7186c4e8688260afb0b7ecfc6",
      "4a05c2c8ff1b4cd9b1a7ef47c5ae767d",
      "ed3723a200a049518170a1da57ba9173",
      "9149dd605d8f4bb99beb6d92f9f807bf",
      "c7220f72ae3a40f0ba9704ef1d5ac33e",
      "8038434f8bc24013932863227058f6d6",
      "fb2dc4c33c5a478aab7fa97bcc7c0bf8",
      "514efa92163f4a92bb1d258676990c10",
      "d145054e94d5408c8b12f3c5e7fe6802",
      "02f52a589ad24dcea0381cc1e7bc78d8",
      "033ad3ff69674168b26de1ec0f77b10f",
      "c3080f8407d84ce0a0f4bdc876803737",
      "373fe27b6d87452c91d59c70cb2c6c35",
      "902d1f76c03c4bd59fb313f6e4236842",
      "a74492a6ac17468f8348abfa394fc4de",
      "cd19a2c43f6347c2895eeca52ecd1afd",
      "555ccd5a93fa4d22a2bb54527a039ed2",
      "2bd298af7c9d4b0d916e0ce3410edf91",
      "b83c4dce70d34e07bba263f91e40df76",
      "7655c6632e3f4a0d9866ede201db3444",
      "f6b2bf12357046d6b87595e747eadc3b",
      "e0a8d3f0d32b4964b8bfdab894324ba8",
      "73510ca287cd4ca3ac6b9107d6bfaabf",
      "a2032f0fbd894283b0ddc6ab449b1799",
      "fdd33a23f97c458db3fccfbe69020c23",
      "6834ab2aef16478e841a25bdb7074e98",
      "89ef346e21024178a85dbe60713e6db6",
      "eb9874a0b51947cc8ef2230dd9449d20",
      "15d80b9a70be488ba64ed0be540dd291",
      "511fc1990a464e6da12247f440f475d0",
      "ea2aec8a76504f119abd2031dacec1d9",
      "8e9a9b41fb9e48a2be2abc7c70d0a3b4",
      "457b2a057955411b9dad8c959c03ee57",
      "f4ea03e0bfa6435285e79c67be84aba4",
      "ddf25e41164942ada41849e37b7ac669",
      "0a6dd037f3eb474d9d24f48cd3aaa07c",
      "89cafafadaf24194b954b19a61967419",
      "a4bf98abbfe04c628a51867f28bd1f72",
      "7044936b334f4efe9dc00f07dd795f6d",
      "8d76013074d64af3a3afdc27d96957d5",
      "c65c8ea4d62d4e3d83a9e0508a471dbf",
      "d8343ad35fe7413680395e4c6e33f5e3",
      "f01556078c1e458f80dd4a7fb9f4333c",
      "ce5b8f5c30314c3d8c36685b7a0cb4a0",
      "da104ca845b740d9b97e17fb3434be54",
      "bca0dbac092c4135bb9ccc7898df2813",
      "883341a08a63438591530d4f7a3eabc2",
      "1f50f750f25e425b9b55bb5a4611a529",
      "a8e84c2a9dc84c3aba2bb0856b69ca2f",
      "5b85ef54de2c4909b37863680ab5cb93",
      "ab3257f86f8940678153dd204dfa0313",
      "a62364818ccd4d92bc9a50f7e2473b2a",
      "5a3f247cacd8416589346ecf5a75c041",
      "64f01f4810874425b3331c793551a7b3",
      "b91c69f3dd34440daf6b78fa084abcab",
      "96a87294c75e41a88271623294cd2065",
      "f76531b509e443a888b28e6260d22bf4",
      "ed1deff05e9d4e8db50ba4561ee41999",
      "83cdbcbc48d9425cbb286268a92aaa63",
      "03358fd83a91437da79b071577016390",
      "ee64b259424d488db2309395b23a4e48",
      "4dfdbf9da6c74164837c13f25d8db51e",
      "3bb45b017f5f4cc9b70c302512dfc1c4",
      "6bb73606090e4ed8b8a8be3e8d840ca3",
      "295ea4c6dc0c4db9a5e46673e39fe551",
      "03953db0528340c183ae888d06d6d7ef"
     ]
    },
    "id": "Bv14vmbUT87v",
    "outputId": "b161a57a-4acd-4be7-ab65-9bb55f59edff"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.10/dist-packages/huggingface_hub/utils/_token.py:89: UserWarning: \n",
      "The secret `HF_TOKEN` does not exist in your Colab secrets.\n",
      "To authenticate with the Hugging Face Hub, create a token in your settings tab (https://huggingface.co/settings/tokens), set it as secret in your Google Colab and restart your session.\n",
      "You will be able to reuse this secret in all of your notebooks.\n",
      "Please note that authentication is recommended but still optional to access public models or datasets.\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "dcaa0ac7186c4e8688260afb0b7ecfc6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "tokenizer_config.json:   0%|          | 0.00/366 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c3080f8407d84ce0a0f4bdc876803737",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "vocab.txt:   0%|          | 0.00/232k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "73510ca287cd4ca3ac6b9107d6bfaabf",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "tokenizer.json:   0%|          | 0.00/711k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f4ea03e0bfa6435285e79c67be84aba4",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "special_tokens_map.json:   0%|          | 0.00/125 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "da104ca845b740d9b97e17fb3434be54",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "config.json:   0%|          | 0.00/743 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "96a87294c75e41a88271623294cd2065",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "model.safetensors:   0%|          | 0.00/133M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
    "embed_model = (\n",
    "    get_registry()\n",
    "    .get(\"huggingface\")\n",
    "    .create(name=\"BAAI/bge-small-en-v1.5\", device=device)\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "CjXFZnmwT87w"
   },
   "source": [
    "# LanceDB Configuration"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "brSsW44RT87y"
   },
   "source": [
    "In this process, we'll create a vector store by defining a schema that includes text data and their corresponding embedding vectors. We'll set up a class, Vectorstore, which initializes with context information and a database path, establishes a connection to LanceDB, and persists the context data into a database table if it doesn't already exist. Additionally, we'll implement a method to search this table using hybrid queries, retrieving and ranking the most relevant context blocks based on the input query. This setup enables efficient storage, retrieval, and querying of contextual data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "id": "SvOk55xqT87z"
   },
   "outputs": [],
   "source": [
    "class Schema(LanceModel):\n",
    "    text: str = embed_model.SourceField()\n",
    "    vector: Vector(embed_model.ndims()) = embed_model.VectorField()\n",
    "\n",
    "\n",
    "class Vectorstore:\n",
    "    def __init__(\n",
    "        self, context_information=None, db_path=None, tablename=\"context\", chunk_size=50\n",
    "    ):\n",
    "        if context_information is None or db_path is None:\n",
    "            raise ValueError(\"Both context_information and db_path must be provided\")\n",
    "\n",
    "        self.context_information = context_information\n",
    "        self.db_path = db_path\n",
    "        self.tablename = tablename\n",
    "        self.chunk_size = chunk_size\n",
    "        self.device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
    "        self.embed_model = (\n",
    "            get_registry()\n",
    "            .get(\"huggingface\")\n",
    "            .create(name=\"BAAI/bge-small-en-v1.5\", device=self.device)\n",
    "        )\n",
    "        self.db = lancedb.connect(self.db_path)\n",
    "        self._persist_on_db()\n",
    "\n",
    "    def split_text_into_chunks(self):\n",
    "        \"\"\"Splits the context information into chunks of the specified size.\"\"\"\n",
    "        words = self.context_information.split()\n",
    "        return [\n",
    "            \" \".join(words[i : i + self.chunk_size])\n",
    "            for i in range(0, len(words), self.chunk_size)\n",
    "        ]\n",
    "\n",
    "    def _persist_on_db(self):\n",
    "        if self.tablename not in self.db.table_names():\n",
    "            tbl = self.db.create_table(self.tablename, schema=Schema, mode=\"overwrite\")\n",
    "            contexts = [\n",
    "                {\"text\": re.sub(r\"\\s+\", \" \", text)}\n",
    "                for text in self.split_text_into_chunks()\n",
    "            ]\n",
    "            tbl.add(contexts)\n",
    "        else:\n",
    "            tbl = self.db.open_table(self.tablename)\n",
    "\n",
    "        tbl.create_fts_index(\"text\", replace=True)\n",
    "\n",
    "    def search_table(self, query_string, query_type=\"hybrid\", top_k=3):\n",
    "        print(\n",
    "            f\"Searching table with query type: {query_type}, table: {self.tablename}, query: {query_string}\"\n",
    "        )\n",
    "        reranker = LinearCombinationReranker(weight=0.7)\n",
    "        tbl = self.db.open_table(self.tablename)\n",
    "        rs = (\n",
    "            tbl.search(query_string, query_type=query_type)\n",
    "            .rerank(reranker=reranker)\n",
    "            .limit(top_k)\n",
    "            .to_list()\n",
    "        )\n",
    "        return \"- context block: \".join([item[\"text\"] for item in rs])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "9oj866vUT871"
   },
   "source": [
    "# DSPy Configuration\n",
    "\n",
    "We'll begin by configuring DSPy with a language model to handle our natural language processing tasks. Here, we use an OpenAI model, specifically the `gpt-4o-mini`, to power our language model (LLM) within DSPy. This setup is flexible—while we're using an OpenAI model in this instance, it's also possible to run any local LLM that is compatible with DSPy. By using a tool like Ollama, you can easily switch to a local LLM by modifying the model configuration. This approach allows for adaptability depending on your computational resources or specific model preferences.\n",
    "\n",
    "All models and comprehensive documentation for DSPy, including how to configure and use different language models, can be found [here](https://dspy-docs.vercel.app/docs/building-blocks/language_models).\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "id": "oZ5ggjwcT873"
   },
   "outputs": [],
   "source": [
    "llm = dspy.OpenAI(\n",
    "    model=\"gpt-4o-mini\",\n",
    ")\n",
    "dspy.configure(lm=llm)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "BrAa-QYHT873"
   },
   "source": [
    "In DSPy, a signature defines the structure and expected inputs and outputs for a task, serving as a blueprint that ensures consistency in data handling and task execution. By clearly specifying the inputs, outputs, and their types, signatures help maintain a standardized approach to implementing various tasks within your pipeline. This ensures that different components can interact seamlessly and that the data flows correctly through each step of the process.\n",
    "\n",
    "For more detailed information about signatures and how to use them, you can explore the [DSPy documentation on signatures](https://dspy-docs.vercel.app/docs/building-blocks/signatures)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "id": "P33U3gPCT874"
   },
   "outputs": [],
   "source": [
    "class GenerateAnswer(dspy.Signature):\n",
    "    \"\"\"As a chat assitant, generates an answer based on a query and given context chunks.\"\"\"\n",
    "\n",
    "    query = dspy.InputField(\n",
    "        desc=\"The question or query to be answered, if context is not provided answers respectfully that cannot help with that question\",\n",
    "        type=str,\n",
    "    )\n",
    "    context_chunks = dspy.InputField(\n",
    "        desc=\"List of relevant context chunks to answer the query\", type=list\n",
    "    )\n",
    "    answer = dspy.OutputField(desc=\"The answer to the query, 5-20 words\")\n",
    "    answer_rationale = dspy.OutputField(\n",
    "        desc=\"LM's reasoning before it generates the output\"\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "58nzIKRMT875"
   },
   "source": [
    "We're implementing a Retrieval-Augmented Generation (RAG) module, which is a method that enhances the generation of answers by retrieving relevant information from a knowledge base.\n",
    "\n",
    "RAG works by first searching for relevant context using a vector-based search in LanceDB, a high-performance database optimized for storing and querying multi-modal data. The vectorstore in LanceDB enables efficient retrieval of context by matching the user's query with relevant chunks of information stored as vectors. Once the relevant context is retrieved, it is used to generate a more accurate and informed answer.\n",
    "\n",
    "The RAG class integrates these steps: it retrieves context using the LanceDB Vectorstore and then generates the final answer using the ChainOfThought mechanism with the GenerateAnswer signature. This approach ensures that the model provides answers that are both contextually relevant and coherent, leveraging the power of vector-based search for precise and efficient information retrieval."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "id": "hhMZadbqT875"
   },
   "outputs": [],
   "source": [
    "class RAG(dspy.Module):\n",
    "    def __init__(self, context_information, db_path):\n",
    "        super().__init__()\n",
    "        if context_information is None or db_path is None:\n",
    "            raise ValueError(\"Both context_information and db_path must be provided\")\n",
    "\n",
    "        self.vectorstore = Vectorstore(\n",
    "            context_information=context_information, db_path=db_path\n",
    "        )\n",
    "        self.generate_answer = dspy.ChainOfThought(\n",
    "            GenerateAnswer\n",
    "        )  # Using signature defined above\n",
    "\n",
    "    def forward(self, query):\n",
    "        relevant_contexts = self.vectorstore.search_table(\n",
    "            query_string=query, query_type=\"hybrid\", top_k=3\n",
    "        )\n",
    "        prediction = self.generate_answer(query=query, context_chunks=relevant_contexts)\n",
    "\n",
    "        return dspy.Prediction(\n",
    "            query=query,\n",
    "            context_chunks=relevant_contexts,\n",
    "            answer=prediction.answer,\n",
    "            answer_rationale=prediction.answer_rationale,\n",
    "        )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "myW6fqmbT876"
   },
   "source": [
    "# EvaluatorRAG Module"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "n3FXvZwjT876"
   },
   "source": [
    "The EvaluateAnswer class defines a signature for evaluating the accuracy of an answer. It specifies the inputs and outputs necessary to assess the quality of the generated response. The evaluation considers the original query, the context chunks used to form the answer, the answer itself, and the rationale behind it. The output includes an accuracy metric (rated from 0 to 10) and a rationale metric, which provides insight into the reasoning process."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "id": "AXZ9DlXKT876"
   },
   "outputs": [],
   "source": [
    "class EvaluateAnswer(dspy.Signature):\n",
    "    \"\"\"Returns a 0-10 metric that measures the accuracy of the provided answer based on the given context chunks and the rationale provided by the algorithm.\"\"\"\n",
    "\n",
    "    query = dspy.InputField(desc=\"The question or query to be answered.\", type=str)\n",
    "    context_chunks = dspy.InputField(\n",
    "        desc=\"List of relevant context chunks used to answer the query.\", type=list\n",
    "    )\n",
    "    answer = dspy.InputField(desc=\"The provided answer to the query.\", type=str)\n",
    "    answer_rationale = dspy.InputField(\n",
    "        desc=\"The reasoning given by the language model for the answer.\", type=str\n",
    "    )\n",
    "    accuracy_metric = dspy.OutputField(\n",
    "        desc=\"0-10 number that represents a metric evaluating the accuracy of the answer based on the answer vs contexts provided\",\n",
    "        type=int,\n",
    "    )\n",
    "    rationale_metric = dspy.OutputField(desc=\"LM's metric reasoning\", type=str)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "5oik0DN4T877"
   },
   "source": [
    "The EvaluatorRAG class is a module designed to implement the evaluation process defined by the EvaluateAnswer signature. It initializes the evaluation mechanism and provides a method (forward) that takes in the query, context chunks, the generated answer, and its rationale. This method evaluates the accuracy of the answer and normalizes the resulting accuracy metric to ensure it's a usable number. The module then returns a prediction object that includes the original inputs along with the evaluated accuracy and rationale metrics, providing a comprehensive assessment of the answer's quality."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "id": "CX27u6BJT877"
   },
   "outputs": [],
   "source": [
    "class EvaluatorRAG(dspy.Module):\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        self.evaluate_answer = dspy.ChainOfThought(\n",
    "            EvaluateAnswer\n",
    "        )  # Using the EvaluateAnswer signature\n",
    "\n",
    "    def forward(self, query, context_chunks, answer, answer_rationale):\n",
    "        evaluation = self.evaluate_answer(\n",
    "            query=query,\n",
    "            context_chunks=context_chunks,\n",
    "            answer=answer,\n",
    "            answer_rationale=answer_rationale,\n",
    "        )\n",
    "\n",
    "        # Normalize the accuracy metric to ensure it's always a number\n",
    "        accuracy_metric = self.normalize_metric(evaluation.accuracy_metric)\n",
    "\n",
    "        return dspy.Prediction(\n",
    "            query=query,\n",
    "            context_chunks=context_chunks,\n",
    "            answer=answer,\n",
    "            answer_rationale=answer_rationale,\n",
    "            accuracy_metric=accuracy_metric,\n",
    "            rationale_metric=evaluation.rationale_metric,\n",
    "        )\n",
    "\n",
    "    def normalize_metric(self, metric):\n",
    "        if isinstance(metric, str):\n",
    "            match = re.search(r\"\\d+\", metric)\n",
    "            if match:\n",
    "                return int(match.group())\n",
    "        return metric"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "JoZgA3RFT878"
   },
   "source": [
    "# RAG_Assitant Module\n",
    "The RAG_Assitant class encapsulates both the generation and evaluation of answers within a single chain of operations. It initializes the RAG module for retrieving and generating the answer, and the EvaluatorRAG module for assessing the quality of that answer.\n",
    "\n",
    "In the process_question method, the class first processes the query using the RAG module to generate an answer and relevant context. This result is then passed to the EvaluatorRAG module, which evaluates the accuracy and reasoning behind the generated answer. The final output is a comprehensive dictionary that includes the query, the context chunks used, the generated answer, the reasoning behind the answer, and the evaluation metrics. This structured approach ensures that both the generation and evaluation steps are seamlessly integrated, providing a robust solution for answering and assessing queries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "id": "hWQTfpwpT878"
   },
   "outputs": [],
   "source": [
    "class RAG_Assitant(dspy.Module):\n",
    "    def __init__(self, context_information, db_path):\n",
    "        super().__init__()\n",
    "        self.rag = RAG(context_information=context_information, db_path=db_path)\n",
    "        self.evaluator_rag = EvaluatorRAG()\n",
    "\n",
    "    def process_question(self, query):\n",
    "        # Get the initial result from RAG\n",
    "        result = self.rag.forward(query)\n",
    "\n",
    "        # Evaluate the result using EvaluatorRAG\n",
    "        evaluation = self.evaluator_rag.forward(\n",
    "            query=query,\n",
    "            context_chunks=result.context_chunks,\n",
    "            answer=result.answer,\n",
    "            answer_rationale=result.answer_rationale,\n",
    "        )\n",
    "\n",
    "        # Return the evaluation results as a dictionary\n",
    "        return {\n",
    "            \"query\": evaluation.query,\n",
    "            \"context_chunks\": evaluation.context_chunks,\n",
    "            \"answer\": evaluation.answer,\n",
    "            \"answer_rationale\": evaluation.answer_rationale,\n",
    "            \"accuracy_metric\": evaluation.accuracy_metric,\n",
    "            \"rationale_metric\": evaluation.rationale_metric,\n",
    "        }"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "bWPkTyBLT879"
   },
   "source": [
    "# Testing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "iJlFUpvyT879"
   },
   "source": [
    "### Define context for initialization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "id": "nhN13_mzT879"
   },
   "outputs": [],
   "source": [
    "CONTEXT_DATA = \"\"\"\n",
    "Welcome to BeatyPets\n",
    "Welcome to BeatyPets!\n",
    "Your go-to store for all your pet care needs.\n",
    "Located in Petville, PA, BeatyPets offers a variety of services for your furry friends.\n",
    "From grooming and health check-ups to training and boarding, we are here to cater to your pets' needs.\n",
    "BeatyPets is your trusted pet shop, providing comprehensive services for dogs, cats, birds, reptiles, and small mammals. We specialize in grooming, health check-ups, vaccinations, training, and boarding services. Our expert staff includes veterinarians, groomers, and trainers dedicated to the well-being of your pets.\n",
    "Visit us at 123 Pet Lane, Petville, PA 12345. Contact us at 555-123-4567 or email us at contact@beatypets.com. Explore more at http://beatypets.com.\n",
    "Authors: BeatyPets Team\n",
    "Categories: Pets, Pet Care\n",
    "Published: 20240709\n",
    "Updated: 20240709\n",
    "Source: http://beatypets.com\n",
    "\n",
    "BeatyPets Business Hours\n",
    "Business Hours:\n",
    "Monday - Friday: 9 AM - 6 PM\n",
    "Saturday: 10 AM - 4 PM\n",
    "Sunday: Closed\n",
    "BeatyPets operates from Monday to Saturday with varied hours. We are closed on Sundays. Our convenient hours ensure we are available when you need us for your pet care needs.\n",
    "Authors: BeatyPets Team\n",
    "Categories: Pets, Pet Care, Business Hours\n",
    "Published: 20240709\n",
    "Updated: 20240709\n",
    "Source: http://beatypets.com/hours\n",
    "\n",
    "Popular Dog Breeds Served at BeatyPets\n",
    "Popular Dog Breeds Served:\n",
    "Labrador Retriever\n",
    "French Bulldog\n",
    "German Shepherd\n",
    "Golden Retriever\n",
    "Bulldog\n",
    "At BeatyPets, we serve a variety of popular dog breeds, including Labrador Retrievers, French Bulldogs, German Shepherds, Golden Retrievers, and Bulldogs. Our services cater to the needs of these beloved breeds with specialized care.\n",
    "Authors: BeatyPets Team\n",
    "Categories: Pets, Dog Breeds\n",
    "Published: 20240709\n",
    "Updated: 20240709\n",
    "Source: http://beatypets.com/popular-dogs\n",
    "\n",
    "BeatyPets Cat Policy\n",
    "Cat Policy:\n",
    "You can bring your cat to BeatyPets! We provide grooming and health check-up services specifically for cats.\n",
    "At BeatyPets, we welcome cats and offer specialized services including grooming and health check-ups. Your cat is in good hands with our expert staff, who ensure their comfort and well-being.\n",
    "Authors: BeatyPets Team\n",
    "Categories: Pets, Cat Care\n",
    "Published: 20240709\n",
    "Updated: 20240709\n",
    "Source: http://beatypets.com/cat-policy\n",
    "\n",
    "Meet Our Staff at BeatyPets\n",
    "Meet Our Staff:\n",
    "Dr. Sarah Johnson - Veterinarian, Small Animals\n",
    "Michael Brown - Groomer, Dogs and Cats\n",
    "Emily Davis - Trainer, Behavioral Training\n",
    "Meet our dedicated staff at BeatyPets who are committed to providing exceptional care for your pets. Our team includes experienced professionals in veterinary care, grooming, and training.\n",
    "Authors: BeatyPets Team\n",
    "Categories: Pets, Staff\n",
    "Published: 20240709\n",
    "Updated: 20240709\n",
    "Source: http://beatypets.com/staff\n",
    "\n",
    "Upcoming Events at BeatyPets\n",
    "Upcoming Events:\n",
    "Pet Adoption Day - August 15, 2024\n",
    "Pet Health Seminar - September 10, 2024\n",
    "Join us at BeatyPets for exciting upcoming events! From pet adoption days to health seminars, there's something for every pet owner. Mark your calendar and don't miss out!\n",
    "Authors: BeatyPets Team\n",
    "Categories: Pets, Events\n",
    "Published: 20240709\n",
    "Updated: 20240709\n",
    "Source: http://beatypets.com/events\n",
    "\n",
    "Appointments Available at BeatyPets\n",
    "Appointments Available:\n",
    "Monday - Friday: 9 AM - 6 PM\n",
    "Saturday: 10 AM - 4 PM\n",
    "Sunday: Closed\n",
    "Services requiring an appointment: grooming, veterinary check-ups, training.\n",
    "BeatyPets offers appointments from Monday to Saturday for services such as grooming, veterinary check-ups, and training. We are closed on Sundays. Be sure to book your appointment in advance to secure availability.\n",
    "Authors: BeatyPets Team\n",
    "Categories: Pets, Pet Care, Appointments\n",
    "Published: 20240709\n",
    "Updated: 20240709\n",
    "Source: http://beatypets.com/appointments\n",
    "\n",
    "Products for Sale at BeatyPets\n",
    "Products for Sale:\n",
    "- Pet toys\n",
    "- Beds and crates\n",
    "- Collars and leashes\n",
    "- Hygiene products\n",
    "- Food and treats\n",
    "At BeatyPets, we offer a wide range of pet products, including toys, beds, crates, collars, leashes, hygiene products, and a variety of food and treats. Find everything your pet needs in one place.\n",
    "Authors: BeatyPets Team\n",
    "Categories: Pets, Products\n",
    "Published: 20240709\n",
    "Updated: 20240709\n",
    "Source: http://beatypets.com/products\n",
    "\n",
    "Types of Pet Food at BeatyPets\n",
    "Types of Pet Food:\n",
    "- Dry food\n",
    "- Wet food\n",
    "- Dietary food\n",
    "- Grain-free food\n",
    "- Treats and snacks\n",
    "BeatyPets offers a variety of pet foods, including dry, wet, dietary, and grain-free options. We also have a selection of treats and snacks to keep your pets happy.\n",
    "Authors: BeatyPets Team\n",
    "Categories: Pets, Food\n",
    "Published: 20240709\n",
    "Updated: 20240709\n",
    "Source: http://beatypets.com/food\n",
    "\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "XuDwjOjQT87_"
   },
   "source": [
    "## Let's create our assitant\n",
    "This line of code initializes the RAG_Assitant module by passing in the necessary context information and database path. The context_information parameter, provided as CONTEXT_DATA, contains the data that will be used to generate and evaluate answers. The db_path parameter specifies the path to the LanceDB database where the context data is stored and managed. This initialization prepares the RAG_Assitant for processing queries by setting up the entire chain from context retrieval to answer evaluation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "id": "1pY7kH6KT87_"
   },
   "outputs": [],
   "source": [
    "assistant = RAG_Assitant(context_information=CONTEXT_DATA, db_path=\"./db_lancedb\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "id": "ufjpkulHecw3"
   },
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "os.environ[\"OPENAI_API_KEY\"] = \"sk-proj-...\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "2oQIhfT7T87_",
    "outputId": "e2894b77-c893-466e-8f70-b0f62e7b9afa"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Searching table with query type: hybrid, table: context, query: Is it open on Tuesday?\n",
      "Answer: Yes, BeatyPets is open on Tuesday.\n",
      "Answer Rationale: The context confirms that BeatyPets operates Monday to Friday, making it open on Tuesday.\n",
      "Accuracy: 10\n",
      "Accuracy Rationale: The answer is fully supported by the context, confirming that BeatyPets is indeed open on Tuesday.\n"
     ]
    }
   ],
   "source": [
    "test_question = \"Is it open on Tuesday?\"\n",
    "result = assistant.process_question(test_question)\n",
    "print(\"Answer:\", result[\"answer\"])\n",
    "print(\"Answer Rationale:\", result[\"answer_rationale\"])\n",
    "print(\"Accuracy:\", result[\"accuracy_metric\"])\n",
    "print(\"Accuracy Rationale:\", result[\"rationale_metric\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "vaMoIxVUT88A",
    "outputId": "ade29e8f-b809-494f-d57f-8facef212854"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Searching table with query type: hybrid, table: context, query: Who is the veterinarian at BeatyPets?\n",
      "Answer: Dr. Sarah Johnson is the veterinarian at BeatyPets.\n",
      "Answer Rationale: The context clearly identifies Dr. Sarah Johnson as the veterinarian, allowing for a straightforward answer.\n",
      "Accuracy: 10\n",
      "Accuracy Rationale: The answer is fully supported by the context, and the rationale correctly explains the basis for the answer.\n"
     ]
    }
   ],
   "source": [
    "test_question = \"Who is the veterinarian at BeatyPets?\"\n",
    "result = assistant.process_question(test_question)\n",
    "print(\"Answer:\", result[\"answer\"])\n",
    "print(\"Answer Rationale:\", result[\"answer_rationale\"])\n",
    "print(\"Accuracy:\", result[\"accuracy_metric\"])\n",
    "print(\"Accuracy Rationale:\", result[\"rationale_metric\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "CyPLyzQTT88B"
   },
   "source": [
    "### Challenging questions\n",
    "We are undertaking a series of challenging questions to rigorously test different metrics within our RAG and evaluation modules. This process will help ensure that the models are not only generating accurate responses but are also providing well-reasoned justifications and maintaining high standards of evaluation accuracy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "RhzfP7kvT88B",
    "outputId": "d642d1a8-f5ed-4bea-d239-fcd52b2217b9"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Searching table with query type: hybrid, table: context, query: Can BeatyPets handle aggressive dogs?\n",
      "Answer: The context does not specify handling aggressive dogs.\n",
      "Answer Rationale: The reasoning process involved analyzing the context provided, which focused on the breeds served and staff expertise but lacked specific information about handling aggressive dogs.\n",
      "Accuracy: 9\n",
      "Accuracy Rationale: The answer accurately identifies the lack of information regarding aggressive dogs in the context, demonstrating a clear understanding of the provided material.\n"
     ]
    }
   ],
   "source": [
    "test_question = \"Can BeatyPets handle aggressive dogs?\"\n",
    "result = assistant.process_question(test_question)\n",
    "print(\"Answer:\", result[\"answer\"])\n",
    "print(\"Answer Rationale:\", result[\"answer_rationale\"])\n",
    "print(\"Accuracy:\", result[\"accuracy_metric\"])\n",
    "print(\"Accuracy Rationale:\", result[\"rationale_metric\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "bIrNVp50T88C",
    "outputId": "bb0fd456-cdb6-417f-e4dd-756e5b0cda0c"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Searching table with query type: hybrid, table: context, query: Can I schedule a grooming appointment online?\n",
      "Answer: It's unclear if you can schedule online.\n",
      "Answer Rationale: The context mentions appointments are needed but does not specify if they can be scheduled online.\n",
      "Accuracy: 7\n",
      "Accuracy Rationale: The answer captures the uncertainty present in the context but could be improved by acknowledging\n"
     ]
    }
   ],
   "source": [
    "test_question = \"Can I schedule a grooming appointment online?\"\n",
    "result = assistant.process_question(test_question)\n",
    "print(\"Answer:\", result[\"answer\"])\n",
    "print(\"Answer Rationale:\", result[\"answer_rationale\"])\n",
    "print(\"Accuracy:\", result[\"accuracy_metric\"])\n",
    "print(\"Accuracy Rationale:\", result[\"rationale_metric\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "Txwg2gnnT88C",
    "outputId": "edb05d27-b119-4e81-c022-3abb9889f107"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Searching table with query type: hybrid, table: context, query: Is BeatyPets open on public holidays?\n",
      "Answer: BeatyPets is likely closed on public holidays.\n",
      "Answer Rationale: The reasoning is based on the provided context, which states that BeatyPets is closed on Sundays and does not mention any special hours for public holidays.\n",
      "Accuracy: 9\n",
      "Accuracy Rationale: The answer is mostly accurate, but the use of \"likely\" introduces a slight uncertainty that is not explicitly supported by the context. A definitive statement would have\n"
     ]
    }
   ],
   "source": [
    "test_question = \"Is BeatyPets open on public holidays?\"\n",
    "result = assistant.process_question(test_question)\n",
    "print(\"Answer:\", result[\"answer\"])\n",
    "print(\"Answer Rationale:\", result[\"answer_rationale\"])\n",
    "print(\"Accuracy:\", result[\"accuracy_metric\"])\n",
    "print(\"Accuracy Rationale:\", result[\"rationale_metric\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "c3r3LAh0T88D"
   },
   "source": [
    "# Conclusions\n",
    "The BeatyPets RAG System represents a sophisticated application of modern AI techniques, blending advanced natural language processing with robust data management to create an intelligent, responsive system. By leveraging DSPy for seamless integration and task management, along with LanceDB for efficient vector-based data retrieval, the system is able to deliver precise, context-aware answers to user queries.\n",
    "\n",
    "This project not only demonstrates the power of combining state-of-the-art tools like DSPy and LanceDB but also provides a flexible framework that can be adapted to a wide range of domains beyond pet care. Whether used for customer support, virtual assistants, or knowledge management, the principles and architecture of this system offer a solid foundation for building intelligent, scalable, and user-friendly applications.\n",
    "\n",
    "As AI continues to evolve, projects like this underscore the importance of integrating multiple technologies to achieve superior performance and usability. The BeatyPets RAG System is a testament to the potential of AI in enhancing user experiences through intelligent, contextually aware interactions."
   ]
  }
 ],
 "metadata": {
  "colab": {
   "provenance": []
  },
  "kernelspec": {
   "display_name": "venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
